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ABSTRACT 

This  essay,  which  reviews  the  foundations,  extensions,  and  challenges  of  cognitive  work 
analysis,  is  based  on  a  keynote  address  delivered  at  the  10th  International  Naturalistic 
Decision  Making  Conference  held  in  Orlando,  Florida  from  31  May  to  3  June,  2011.  It 
describes  the  origins  of  cognitive  work  analysis  and  the  utility  of  this  framework  for 
designing  ecological  interfaces,  as  well  as  for  tackling  a  variety  of  other  design  challenges, 
particularly,  the  design  of  teams  or  organisations.  Also  featured  in  this  essay  is  the 
formulation  of  a  methodological  perspective  of  cognitive  work  analysis,  which  complements 
the  conceptual  accounts  provided  by  Rasmussen  (1986),  Rasmussen,  Pejtersen,  and  Goodstein 
(1994),  and  Vicente  (1999).  Finally,  this  essay  highlights  the  latest  shift  in  research  emphasis 
from  work  domain  analysis,  the  first  dimension  of  cognitive  work  analysis,  to  the  subsequent 
dimensions  of  this  framework. 
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Cognitive  Work  Analysis: 

Foundations,  Extensions,  and  Challenges 

Executive  Summary 

Workers  in  complex  sociotechnical  systems,  such  as  military  systems,  have  incredibly 
challenging  jobs.  To  help  them  perform  their  jobs  effectively,  it  is  indisputable  that  first 
we  need  to  understand  the  nature  of  their  work.  Only  then  is  it  possible  to  design 
interfaces,  teams,  or  training  systems,  for  instance,  that  will  enable  workers  to  meet 
their  work  demands  successfully  -  in  a  way  that  is  safe,  productive,  and  healthy 
(Vicente,  1999). 

This  essay,  which  is  based  on  a  keynote  address  delivered  at  the  10th  International 
Naturalistic  Decision  Making  Conference  held  in  Orlando,  Florida  from  31  May  to  3 
June,  2011,  reviews  the  foundations,  extensions,  and  challenges  of  cognitive  work 
analysis,  a  framework  for  the  analysis,  design  and  evaluation  of  work  in  complex 
sociotechnical  systems  (Rasmussen,  1986;  Rasmussen,  Pejtersen  &  Goodstein,  1994; 
Vicente,  1999).  This  framework  comprises  five  dimensions  of  analysis,  namely,  work 
domain  analysis,  control  task  analysis,  strategies  analysis,  social  organisation  and 
cooperation  analysis,  and  worker  competencies  analysis.  Each  dimension  is  concerned 
with  the  analysis  of  particular  types  of  constraints  on  actors'  behaviour.  Within  these 
constraints,  actors  have  many  possibilities  for  action.  By  focusing  on  the  analysis  of 
constraints,  rather  than  specific  instances  of  behaviour,  cognitive  work  analysis 
promotes  designing  for  adaptation. 

This  essay  demonstrates  that  there  have  been  substantial  research  outcomes  in 
cognitive  work  analysis  over  the  last  few  decades.  Included  in  these  achievements  is 
the  development  of  the  cognitive  work  analysis  framework  itself,  which  has  its  origins 
in  the  work  of  Jens  Rasmussen  and  his  colleagues  at  the  Riso  National  Laboratory  in 
Denmark.  Based  on  empirical  studies  of  how  work  is  achieved  in  naturalistic  settings, 
these  researchers  conceptualised  a  solid  foundation  for  work  analysis. 

Subsequently,  some  of  the  concepts  of  cognitive  work  analysis  were  extended  into 
ecological  interface  design  (EID),  a  framework  for  designing  interfaces  for  complex 
sociotechnical  systems  (Rasmussen  &  Vicente,  1989;  Vicente  &  Rasmussen,  1990,  1992). 
Unlike  other  approaches  to  interface  design,  the  EID  framework  has  been  subjected  to 
a  comprehensive  program  of  controlled  experimental  investigations.  These 
investigations  have  demonstrated  that  EID  can  lead  to  better  performance  than  existing 
interfaces  for  many  types  of  systems,  including  military  ones.  The  EID  program  has 
addressed  in  part  some  of  the  limitations  of  the  original  research  by  the  Riso  group, 
which  relate  to  the  lack  of  experimental  testing  of  the  concepts  of  cognitive  work 
analysis  and  the  generalisability  of  these  concepts  to  a  variety  of  systems.  Future 
research  on  EID  will  include  extensions  to  this  framework  to  incorporate,  among  other 
things,  additional  concepts  of  cognitive  work  analysis  and  other  work  analysis 
techniques. 
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Novel  applications  of  cognitive  work  analysis,  to  problems  other  than  interface  design, 
have  been  achieved  largely  in  industrial  settings.  These  applications  include  the 
evaluation  of  system  design  concepts,  the  definition  of  training  needs  and  training- 
system  requirements,  and  the  development  of  new  team  or  organisational  designs. 
Some  of  these  applications  have  required  extensions  to  the  basic  concepts  of  cognitive 
work  analysis.  While  these  applications  have  had  an  impact  in  industrial  settings,  and 
have  made  a  unique  contribution  to  design  relative  to  standard  techniques,  empirical 
evaluation  is  lacking.  For  this  to  be  achieved,  a  shift  to  a  research  setting  may  be 
necessary. 

The  methodological  program  on  cognitive  work  analysis  complements  the  seminal 
monographs  by  Rasmussen  (1986),  Rasmussen  et  al.  (1994),  and  Vicente  (1999),  which 
provide  a  comprehensive  conceptual  perspective  of  this  framework.  This  program 
recognises  that  the  widespread  adoption  of  cognitive  work  analysis  is  hampered  by  the 
fact  that  this  technique  for  work  analysis  is  challenging,  not  only  to  learn,  but  also  to 
perform  well.  While  a  substantive  start  has  been  made  on  methodology,  much  more 
work  is  necessary,  especially  in  relation  to  the  dimensions  subsequent  to  work  domain 
analysis. 

The  future  program  on  the  subsequent  dimensions  will  echo  the  historical 
development  of  cognitive  work  analysis  by  building  on  the  research  of  a  range  of 
communities.  Given  its  complexity,  it  is  improbable  that  the  nature  of  human  work  in 
sociotechnical  systems  can  be  properly  studied  or  understood  by  any  one  discipline  or 
community.  By  contributing  carefully  investigated  and  detailed  insights  into  diverse 
aspects  of  human  work,  multiple  communities  can  provide  valuable  leverage  points  for 
cognitive  work  analysis.  A  significant  challenge  for  researchers  will  be  to  build  on  any 
insights  in  a  way  that  remains  faithful  to  the  theoretical  underpinnings  of  cognitive 
work  analysis,  which  appears  to  offer  a  powerful  approach  to  design. 

The  preceding  achievements  in  cognitive  work  analysis  are  likely  to  spark  significant 
interest  in  this  framework  over  the  next  decade.  An  immediate  outcome  of  the  growing 
popularity  of  this  framework  will  be  a  proliferation  of  applications  of  cognitive  work 
analysis,  including  many  novel  applications.  This  situation  will  emphasise  one  of  the 
biggest  challenges  of  cognitive  work  analysis,  specifically,  to  support  an  integrated 
approach  to  system  design. 

Designing  for  adaptation  is  a  complex  problem  that  will  not  be  resolved  in  a  piecemeal 
fashion.  It  will  not  be  resolved  by  just  an  ecological  interface  (Vicente,  1999)  or  just  a 
team  design,  for  instance.  Instead,  all  elements  of  a  system's  design  must  be 
systematically  integrated  in  a  way  that  explicitly  supports  this  objective,  which  raises 
theoretical,  methodological,  and  practical  challenges.  I  believe  that  cognitive  work 
analysis  has  the  potential  to  meet  these  challenges,  but  the  task  will  not  be  simple. 
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1.  Introduction 


A  quarter  of  a  century  has  passed  since  Jens  Rasmussen  published  his  first  seminal 
monograph  on  cognitive  work  analysis  (Rasmussen,  1986).  More  than  a  decade  has  slipped  by 
since  Kim  Vicente's  influential  account  of  the  same  topic  was  published  (Vicente,  1999).  The 
10th  anniversary  of  the  Naturalistic  Decision  Making  Conference1  seemed  a  fitting  occasion  to 
review  the  foundations  of  cognitive  work  analysis,  to  examine  what  has  been  achieved  in  this 
area  of  research,  particularly  over  the  last  decade,  and  to  consider  what  challenges  remain  to 
be  addressed  in  the  future.  Some  may  question,  quite  rightfully,  whether  it  is  appropriate  to 
focus  on  cognitive  work  analysis  at  a  conference  of  the  naturalistic  decision  making 
community.  After  all,  don't  those  in  this  community  mainly  use  cognitive  task  analysis 
methods?  While  this  is  most  likely  true,  the  reason  I  am  convinced  of  its  appropriateness  is 
that  cognitive  work  analysis  is  based  on  naturalistic  observation.  Cognitive  work  analysis  was 
developed  in  keeping  with  observations  of  how  work  is  achieved  in  the  real  world.  This 
theme  underpins  this  essay  on  the  foundations,  extensions,  and  challenges  of  cognitive  work 
analysis. 


2.  Why  Work  Analysis? 

It  is  quite  safe  to  assume,  I  imagine,  that  none  in  the  naturalistic  decision  making  community 
would  doubt  the  importance  of  work  analysis;  by  work  analysis  I  mean  any  technique  or 
approach  for  analysing  work  or  tasks.  It  is  unquestionable  that  jobs  in  complex  sociotechnical 
systems,  such  as  hospitals,  military  aviation,  and  emergency  management  organisations,  are 
highly  challenging.  It  also  follows  logically  that  to  support  workers  in  these  demanding  roles, 
it  is  first  necessary  to  understand  the  nature  of  their  work.  Only  then  is  it  possible  to  create 
designs  of  interfaces,  teams,  or  training  systems,  for  example,  that  will  enable  workers  to  meet 
their  work  demands  successfully  -  in  a  way  that  is  safe,  productive,  and  healthy  (Vicente, 
1999). 

There  are  countless  techniques  for  work  analysis,  too  numerous  to  mention  here.  Hierarchical 
task  analysis  (e.g..  Shepherd,  2001),  cognitive  task  analysis  (e.g.,  Schraagen,  Chipman  & 
Shalin,  2000),  and  cognitive  work  analysis  are  some  well  known  ones.  In  this  essay,  I  focus  on 
just  one  of  these  techniques,  specifically,  cognitive  work  analysis.  This  framework  is 
particularly  well  suited  for  the  analysis,  design,  and  evaluation  of  work  in  complex 
sociotechnical  systems. 

3.  What  is  Cognitive  Work  Analysis? 

Cognitive  work  analysis  is  a  framework  that  defines  the  work  demands  of  complex 
sociotechnical  systems  in  terms  of  the  constraints  on  actors  (Rasmussen,  1986;  Rasmussen, 
Pejtersen  &  Goodstein,  1994;  Vicente,  1999).  Many  different  disciplines  use  the  term 
constraints.  For  example,  this  term  features  in  the  language  of  mathematicians  and  engineers. 

1  This  essay  is  based  on  a  keynote  address  delivered  at  the  10th  International  Naturalistic  Decision  Making 
Conference,  Orlando,  FL,  May  31  -  June  3,  2011. 
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However,  this  term  has  different  meanings  in  different  contexts.  In  the  case  of  cognitive  work 
analysis,  constrain  ts  are  limits  on  behaviour,  which  must  be  respected  by  actors  for  a  system  to 
perform  effectively  (Figure  1).  Constraints  distinguish  behaviours  that  are  possible  or 
acceptable  from  those  that  are  impossible  or  unacceptable.  Cognitive  work  analysis,  therefore, 
is  concerned  with  constraints  that  are  "behavior-shaping"  (Rasmussen  et  al.,  1994,  p.  25). 


=  limits  on 
behaviour 


Figure  1  Constraints  place  limits  on  behaviour  but  still  afford  actors  many  possibilities  for  action 

Although  constraints  place  limits  on  behaviour,  actors  still  have  many  options  or  possibilities 
for  action,  as  indicated  by  the  trajectories  (arrows)  in  the  centre  of  Figure  1.  For  instance,  there 
are  many  different  sequences  in  which  actors  can  execute  tasks  or  many  different  strategies 
that  actors  can  use  for  decision  making,  while  remaining  within  the  boundaries  of  effective 
performance. 

To  offer  a  simple  illustration,  a  system's  purposes  and  physical  resources  place  constraints  on 
actors'  behaviour.  It  will  always  be  unacceptable  to  engage  in  actions  that  are  inconsistent 
with  a  system's  purposes  and  it  will  always  be  impossible  to  use  physical  resources  to 
perform  behaviours  they  do  not  have  the  functionality  to  support.  Nevertheless,  within  those 
constraints,  actors  still  have  many  possibilities  for  action  that  are  compatible  with  a  system's 
purposes  and  physical  resources. 

Complex  sociotechnical  systems  can  impose  several  kinds  of  constraints  on  actors. 
Accordingly,  cognitive  work  analysis  comprises  a  number  of  dimensions  of  analysis,  which 
focus  on  different  types  of  constraints  (Table  1).  Work  domain  analysis  focuses  on  constraints 
imposed  by  the  physical,  social,  or  cultural  environment  of  actors,  which  include  a  system's 
purposes  and  physical  resources.  Control  task  analysis  is  concerned  with  constraints  imposed 
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by  the  activity  that  is  necessary  in  a  system.  Strategies  analysis  focuses  on  constraints  imposed 
by  the  strategies  for  performing  an  activity.  Social  organisation  and  cooperation  analysis 
concentrates  on  constraints  associated  with  the  organisation  of  work  in  a  system.  Finally, 
worker  competencies  analysis  is  concerned  with  constraints  associated  with  human  cognitive 
capabilities  and  limitations.  Each  dimension  of  cognitive  work  analysis  has  special  tools  for 
modelling  these  constraints. 

Table  1  The  dimensions  of  cognitive  work  analysis 


Dimension 

Constraints 

Modelling  Tools 

Work  domain 
analysis 

Physical,  social,  or  cultural 
environment,  including 
purposes  &  physical  resources 

Abstraction-decomposition 
space,  abstraction  hierarchy 

Control  task 
analysis 

Work  situations,  work  functions, 
or  control  tasks 

Decision  ladder 

Strategies 

analysis 

Strategies 

Information  flow  map 

Social 

organisation  & 

cooperation 

analysis 

Allocation,  distribution,  or 
coordination  of  work 

All  of  the  above 

Worker 

competencies 

analysis 

Human  cognitive  capabilities 
and  limitations 

Skills,  rules,  and  knowledge 
taxonomy 

Figure  2  shows  that  the  dimensions  of  cognitive  work  analysis  collectively  define  a  constraint- 
based  space,  within  which  actors  have  many  possibilities  for  action.  By  focusing  on 
constraints,  cognitive  work  analysis  promotes  designing  for  adaptation.  Rather  than  developing 
designs  that  support  workers  in  particular  ways  of  working,  which  have  been  prescribed  or 
described  by  analysts  in  advance,  the  aim  is  to  create  designs  that  support  workers  in 
adapting  their  behaviour  as  they  see  fit  without  violating  the  system's  constraints.  With  such 
designs,  workers  can  create  innovative  patterns  of  behaviour,  which  may  be  particularly 
important  for  dealing  with  unforeseen  events,  or  they  can  adopt  ways  of  working  that  suit 
their  individual  preferences,  while  all  the  time  remaining  within  the  boundaries  of  effective 
performance. 
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Worker  competencies 


Social  organisation 
and  cooperation 


work  functions, 
control  tasks 


Figure  2  The  dimensions  of  cognitive  work  analysis  collectively  define  a  constraint-based  space 
within  which  actors  have  many  degrees  of  freedom  for  action. 


4.  Foundations  of  Cognitive  Work  Analysis 


The  cognitive  work  analysis  framework  was  developed  in  the  1960s  and  1970s  (Figure  3).  It 
grew  out  of  a  research  program  at  the  Electronics  Department  of  Riso  National  Laboratory  in 
Denmark,  headed  by  Jens  Rasmussen.  The  aim  of  the  Riso  organisation  at  the  time  was  to 
conduct  research  in  support  of  the  implementation  of  nuclear  power  in  Denmark.  In  this 
section,  an  abbreviated  history  of  the  development  of  cognitive  work  analysis,  based  on 
scientific  publications,  is  provided.  Vicente  (1999, 2001)  offers  a  more  comprehensive  review. 

Timeline 

1960  1970  1980  1990  2000  2010 

I - 1 - 1 - 1 - 1 - 1— ► 

CWA  development  i _ 

Seminal  publications 
Ecological  interface  design 
Seminal  publications 
Beyond  ecological  interface  design 
Methodology 

Beyond  work  domain  analysis 

Figure  3  Timeline  for  key  theoretical  developments  in  cognitive  work  analysis 


J 

e.g.,  Rasmussen,  1983,  1985,  1986 

i _ i 


i _ 

Rasmussen  eta!.,  1994; 
Vicente,  1999  i - 1 
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In  the  early  1960s,  the  focus  of  Rasmussen's  group  was  on  analysing  the  reliability  of  nuclear 
reactor  equipment  and  instrumentation  (Jensen,  Rasmussen  &  Timmerman,  1963;  Rasmussen 
&  Timmerman,  1962).  That  is,  they  were  concerned  with  hardware  reliability.  However,  on  the 
basis  of  empirical  data  collected  in  their  research  facilities,  they  found  that  although  they 
could  design  hardware  systems  with  extremely  high  reliability,  accidents  still  occurred. 

To  try  to  understand  why  this  was  the  case,  an  analysis  of  industrial  accidents  was  conducted 
(Rasmussen,  1968a,  1968b,  1969).  This  study  included  29  cases  in  the  nuclear  domain  and  100 
cases  in  the  air  transportation  domain.  The  analysis  revealed  that  human  error  accounted  for 
approximately  three  quarters  of  these  accidents.  These  errors  arose  when  workers  were 
confronted  with  unfamiliar  situations  that  could  not  have  been  anticipated  by  designers.  As 
these  situations  were  unpredictable,  workers  could  not  have  been  given  pre-planned 
procedures  or  instructions  for  how  to  manage  them  effectively.  However,  the  analysis  also 
showed  that,  in  nearly  all  cases,  workers  could  have  handled  these  situations  satisfactorily  if 
the  actual  state  of  various  aspects  of  the  system  at  the  time  had  been  known  to  them.  These 
findings  highlighted  the  importance  of  designing  to  provide  workers  with  information  about 
the  work  domain,  so  they  can  adapt  their  behaviour  to  deal  with  the  demands  of  a  range  of 
situations,  including  unforeseen  events.  It  is  worth  noting  that,  more  recently,  others  such  as 
Perrow  (1984),  Reason  (1990),  and  Leveson  (1995)  have  also  shown  that  accidents  arise  when 
workers  are  confronted  with  novel  situations. 

In  the  1970s,  the  focus  of  the  Riso  group  shifted  from  the  study  of  machine  reliability  to 
human-machine  reliability.  Accordingly,  these  researchers  conducted  a  number  of  empirical 
studies  with  the  aim  of  establishing  a  sound  basis  for  designing  safer  human-machine 
systems.  Several  of  these  studies  led  to  conceptual  developments,  which  were  ultimately 
integrated  into  the  cognitive  work  analysis  framework.  In  what  follows,  I  describe  one  of  the 
studies  that  contributed  to  the  concepts  and  modelling  tools  of  work  domain  analysis,  the  first 
dimension  of  this  framework.  This  was  a  field  study  of  electronic  troubleshooting. 

In  this  field  study,  Rasmussen  and  Jensen  (1973)  chose  to  investigate  the  way  in  which 
professional  technicians  troubleshoot  faults  in  electronic  equipment.  (This  was  not  a 
surprising  choice  given  that  these  researchers  were  from  an  electronics  department.)  Some  of 
the  aims  of  their  research  were  to  examine  the  problem-solving  strategies  of  workers  and  to 
investigate  the  structure  (i.e.,  event-independent  properties)  of  the  system  that  was  the  object 
of  their  activities.  This  research  would  contribute  to  establishing  how  the  system  can  be 
represented  in  a  model  that  is  both  compatible  with  human  problem-solving  and  event- 
independent.  With  such  a  work  domain  model,  workers  should  be  able  to  reason  about  the 
system  in  a  wide  range  of  situations,  including  unforeseen  events,  and  thus  deal  with  these 
situations  more  effectively. 

The  field  study  included  eight  different  types  of  instruments,  each  with  a  particular  fault,  and 
six  professional  technicians.  The  investigation  was  based  on  verbal  protocol  methodology,  so 
technicians  were  required  to  verbalise  their  problem-solving  processes  as  they  set  about  their 
troubleshooting  tasks.  A  total  of  45  cases  were  recorded  and  transcribed,  although  only  30 
were  subjected  to  detailed  analysis. 
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The  data  analysis  involved  the  development  of  a  preliminary  coding  scheme,  which  was  used 
to  analyse  the  verbal  protocols.  The  protocols  were  then  reviewed  to  determine  if  the 
information  in  them  was  captured  effectively  by  the  coding  scheme.  Any  discrepancies  led  to 
changes  in  the  coding  scheme  until  it  stabilised.  The  verbal  protocols  were  then  analysed  with 
the  final  coding  scheme.  Figure  4  shows  some  of  the  results  of  this  process.  Although  the 
details  are  unclear,  the  illustrations  provide  a  sense  of  the  complexity  of  the  analyses. 


Figure  4  Illustrations  of  the  analyses  performed  by  Rasmussen  and  Jensen  (1973),  reproduced  with 
permission 

On  the  basis  of  the  results  of  this  study,  a  similar  field  study  of  a  conventional  power  plant 
(Rasmussen,  1974),  and  investigations  of  problem  solving  by  other  researchers  including 
Duncker  (1945)  and  de  Groot  (1965),  a  number  of  patterns  were  identified  in  how  workers 
reason  about  complex  systems  during  problem  solving  in  a  range  of  situations  (Figure  5). 
First,  workers  reason  at  different  levels  of  abstraction  while  performing  their  jobs.  Specifically, 
they  spontaneously  shift  their  view  of  a  system  from  purposive  concepts  to  physical  concepts 
in  order  to  match  their  task  demands.  Purposive  concepts  relate  to  information  about  a 
system's  purposes  whereas  physical  concepts  relate  to  information  about  a  system's  physical 
resources. 
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Figure  5  The  abstraction-decomposition  space.  Reprinted  and  adapted  from  Advances  in  man- 
machine  systems  research,  4,  W.  B.  Rouse  (Ed.),  Rasmussen,  ].  A  cognitive  engineering 
approach  to  the  modelling  of  decision  making  and  its  organization  in:  Process  control, 
emergency  management,  CAD/CAM,  office  systems,  and  library  systems  (pp.  165-243), 
Copyright  (1988)  JAI  Press,  with  permission  from  Elsevier. 


Second,  workers  reason  at  different  levels  of  decomposition  while  performing  their  jobs.  That  is, 
they  spontaneously  shift  their  view  of  a  system  from  coarse  levels  of  resolution  to  fine-grained 
levels  of  resolution  in  order  to  match  their  task  demands.  The  most  coarse  level  of  resolution 
relates  to  the  whole  system  whereas  the  most  fine-grained  level  relates  to  its  components. 

Third,  when  the  conceptual  lens  with  which  workers  view  a  system  changes  from  purposive 
models  to  physical  models,  the  level  of  resolution  at  which  they  view  the  system  also  changes 
from  coarse  models  to  fine-grained  models.  As  a  result,  workers  tend  to  adopt  models  along 
the  diagonal  of  this  reasoning  space,  which  indicates  that  these  models  generally  offer  the 
most  meaningful  or  useful  views  of  a  system.  The  numbered  trajectory  in  Figure  5  shows  the 
problem-solving  path  taken  by  an  electronic  technician  in  diagnosing  faults  in  computer 
equipment. 

Fourth,  this  reasoning  space  is  event-independent.  That  is,  although  workers  may  adopt 
diverse  problem-solving  paths  in  the  same  situation,  or  unique  problem-solving  paths  across  a 
range  of  situations,  all  of  these  trajectories  can  be  mapped  onto,  or  can  be  explained  by,  this 
reasoning  space. 
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Based  on  these  findings,  Rasmussen  (1979,  1985)  proposed  the  abstraction-decomposition 
space  as  a  framework  that  can  be  used  to  represent  complex  systems  in  a  way  that  is  both 
consistent  with  the  characteristics  of  human  problem-solving  and  event-independent.  Such  a 
work  domain  model  can  support  human  reasoning  about  complex  systems  in  a  range  of 
situations,  including  those  that  are  novel  or  unforeseen. 

Likewise,  further  empirical  studies  by  the  Riso  group  in  the  1970s  led  to  the  other  concepts 
and  modelling  tools  of  cognitive  work  analysis.  These  include  the  decision  ladder  for  control 
task  analysis,  the  information  flow  map  for  strategies  analysis,  and  the  skills,  rules,  and 
knowledge  taxonomy  for  worker  competencies  analysis  (Table  1).  The  abstraction- 
decomposition  space,  decision  ladder,  and  information  flow  map  are  also  applicable  in  social 
organisation  and  cooperation  analysis. 

In  the  1980s,  Rasmussen  produced  several  seminal  publications  on  cognitive  work  analysis 
(Figure  3).  These  included  journal  papers  on  the  skills,  rules,  and  knowledge  taxonomy 
(Rasmussen,  1983)  and  the  abstraction-decomposition  space  (Rasmussen,  1985).  His  first  book 
(Rasmussen,  1986)  followed. 

Despite  the  achievements  of  the  Riso  program,  there  were  some  limitations  of  this  research,  as 
Vicente  (2001)  recognised.  First,  few  of  the  concepts  of  cognitive  work  analysis,  which  were 
generated  inductively  from  field  studies,  had  been  rigorously  tested  experimentally.  Second,  it 
was  unclear  whether  these  concepts  were  generalisable  to  systems  other  than  those  that  led  to 
their  development.  These  limitations  were  addressed  in  part  during  the  next  major  phase  of 
research  in  the  cognitive  work  analysis  area,  specifically,  ecological  interface  design  (Figure  3). 


5.  Ecological  Interface  Design 

The  ecological  interface  design  (EID)  framework  formalises  some  of  the  concepts  the  Riso 
group  had  been  using  in  designing  interfaces  for  complex  sociotechnical  systems  (Rasmussen 
&  Vicente,  1989;  Vicente  &  Rasmussen,  1990,  1992).  The  primary  aim  of  EID  is  to  support 
worker  adaptation  to  change  and  novelty.  Its  principle  theoretical  foundations  are  the 
abstraction-decomposition  space  and  the  skills,  rules,  and  knowledge  taxonomy. 

To  design  an  ecological  interface,  the  abstraction-decomposition  space  is  used  to  develop  a 
model  of  a  work  domain,  and  thus  define  what  information  is  necessary  on  a  display  to 
support  human  problem-solving  in  that  system.  The  skills,  rules,  and  knowledge  taxonomy  is 
used  to  define  the  form  that  information  should  take  or  how  that  information  should  be 
presented  on  a  display. 

To  provide  an  example  of  a  work  domain  model.  Figure  6  shows  an  abstraction- 
decomposition  space  of  the  human  body,  developed  by  Hajdukiewicz  (1998),  which  may  be 
used  to  create  an  ecological  interface  for  a  surgical  team.  This  model  represents  information 
about  the  human  body  at  different  levels  of  abstraction  and  decomposition,  thereby  defining 
the  information  content  of  a  display. 
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Figure  6  An  abstraction-decomposition  space  of  the  human  body.  Adapted  from  Hajdukiewicz 
(1998),  with  permission  of  the  author. 


The  skills,  rules,  and  knowledge  taxonomy  (Figure  7),  which  is  used  to  define  the  form  of 
information  presentation,  describes  three  qualitatively  different  ways  in  which  people  can 
interact  with  their  environment.  Briefly,  skill-based  behaviour  involves  highly  automated  and 
integrated  patterns  of  action  that  are  directly  coupled  to  the  environment  in  a  continuous 
perception-action  loop.  Walking  or  steering  a  vehicle  are  examples  of  skill-based  behaviour. 
Rule-based  behaviour  is  defined  by  one-to-one  associations  between  familiar  perceptual  cues 
in  the  environment  and  appropriate  actions  or  intentions.  Standard  responses  of  drivers  to 
traffic  lights  are  examples  of  rule-based  behaviour.  Knowledge-based  behaviour  involves 
serial,  analytical  reasoning,  or  problem-solving,  based  on  a  symbolic  mental  representation  of 
the  relevant  constraints  in  the  environment.  For  example,  knowledge-based  behaviour  is 
necessary  for  establishing  the  travel  time  to  an  unfamiliar  destination  based  on  such 
information  as  the  distance  between  the  start  and  end  points,  the  traffic  conditions  at  that  time 
of  day,  and  the  mode  or  means  of  travel. 
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Figure  7  The  skills,  rules,  and  knowledge  taxonomy.  Adapted  from  Rasmussen,  ].  (1983).  Skills, 
rules,  and  knowledge;  signals,  signs,  and  symbols,  and  other  distinctions  in  human 
performance  models.  IEEE  Transactions  on  Systems,  Man,  and  Cybernetics,  13 (3), 
257-266,  (©  1983  IEEE),  with  permission. 

Motivated  by  the  skills,  rules,  and  knowledge  taxonomy,  the  EID  framework  aims  to  present 
work  domain  information  to  workers  in  a  form  that  encourages  the  use  of  skill-based  and 
rule-based  behaviour,  because  these  are  cognitively  less  demanding,  while  still  providing 
support  for  knowledge-based  behaviour,  which  is  necessary  for  dealing  with  novel  events.  To 
achieve  these  aims,  the  EID  framework  comprises  three  design  principles.  To  support  skill- 
based  behaviour,  workers  should  be  able  to  act  directly  on  objects  on  the  display.  To  facilitate 
rule-based  behaviour,  there  should  be  a  one-to-one  mapping  between  work  domain 
constraints  and  perceptual  forms  on  the  display.  To  support  knowledge-based  behaviour,  the 
work  domain  should  be  presented  in  a  form  that  preserves  the  information  and  relationships 
in  the  abstraction-decomposition  space.  This  will  serve  as  an  externalised  mental 
representation  of  work  domain  constraints  to  support  adaptive  problem  solving. 

5.1  Evaluation 

Since  the  EID  framework  was  developed,  a  great  deal  of  research  has  been  conducted  in  this 
area.  In  2002,  Vicente  published  a  review  of  this  body  of  work  in  which  he  discussed  three 
criteria  for  assessing  the  EID  framework.  The  proof  of  principle  criterion  simply  assesses 
whether  the  EID  framework  can  be  applied  to  produce  interfaces  for  complex  sociotechnical 
systems.  The  analytical  criterion  assesses  whether  EID  leads  to  new  inf ormation  requirements 
not  found  in  existing  interfaces.  Lastly,  the  empirical  criterion  assesses  whether  EID  actually 
leads  to  better  performance  compared  with  existing  interfaces. 
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In  this  essay,  I  use  these  three  criteria  to  summarise  the  value  of  the  EID  framework  for 
designing  interfaces  for  complex  sociotechnical  systems.  This  summary  provides  insight  into 
the  validity  and  generalisability  of  some  of  the  concepts  of  cognitive  work  analysis,  as 
established  by  experimental  studies  in  EID.  Accordingly,  this  summary  demonstrates  the 
contribution  of  the  EID  program  to  addressing  some  of  the  limitations  of  the  original  research 
by  the  Riso  group,  which  were  highlighted  earlier.  As  the  EID  literature  is  vast,  I  discuss 
representative  examples  of  studies  that  shed  light  on  these  questions,  rather  than  providing  a 
comprehensive  review. 

As  shown  in  Table  2,  several  studies  of  EID  across  a  range  of  systems  have  tested  this 
framework  against  all  three  criteria,  which  was  not  the  case  when  Vicente  (2002)  published  his 
review.  The  shaded  cells  denote  results  obtained  since  that  review.  I  should  also  note  that,  in 
all  cases  shown  in  this  table,  empirical  evaluation  means  controlled  experimental 
investigations. 

Table  2  Studies  of  EID  across  a  range  of  systems  have  obtained  positive  findings  against  proof  of 

principle  (P-of-P),  analytical,  and  empirical  criteria 


System 

Example  Publications 

P-of-P: 

Applied? 

Analytical: 
New  Info? 

Empirical: 

Perform? 

Process 

Control 

Christoffersen,  Hunter,  & 
Vicente,  1996;  Pawlak& 
Vicente,  1996 

S 

S 

S 

Process 

Control 

Reising  &  Sanderson, 

1998,  2000a, 2000b 

S 

S 

S 

Process 

Control 

Ham  &  Yoon,  2001b 

S 

S 

S 

Process 

Control 

Jamieson,  2007 

S 

S 

S 

Process 

Control 

Lau,  Jamieson,  Skraaning 
Jr.  &  Burns,  2008 

S 

S 

S 

Information 

Retrieval 

Xu,  Dainoff  &  Mark,  1999 

S 

S 

S 

Medicine 

Sharp  &  Helmicki,  1998 

S 

S 

S 

Network 

Man. 

Burns,  Kuo  &  Ng,  2003 

S 

S 

S 

Aviation 

Borst,  Suijkerbuijk,  Mulder 
&  van  Paassen,  2006 

S 

S 

S 

Military 

C&C 

Bennett,  Posey  & 

Shattuck,  2008 
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The  table  shows  that  a  number  of  studies  of  EID  in  the  context  of  process  control  systems  have 
obtained  positive  findings  against  all  three  criteria.  In  other  words,  these  studies  show  that 
EID  can  be  applied  to  process  control  systems,  that  it  does  lead  to  new  information 
requirements  compared  with  existing  interfaces  for  those  systems,  and  that  these  new 
information  requirements  can  lead  to  better  performance  by  workers. 

The  initial  studies  by  Vicente  and  his  colleagues  (e.g.,  Christoffersen  et  al.,  1996;  Pawlak  & 
Vicente,  1996)  were  conducted  with  a  process  control  microworld,  as  were  the  studies  by 
Reising  and  Sanderson  (e.g.,  1998,  2000a,  2000b).  Hence  these  studies  examined  the 
application  of  EID  to  process  control  systems  in  the  context  of  relatively  small-scale  problems. 
Ham  and  Yoon's  (2001b)  work,  however,  which  was  conducted  with  a  nuclear  power  plant 
secondary  cooling  system  simulation,  represented  a  medium-scale  problem.  More  recently, 
Jamieson  (2007)  and  Lau  et  al.  (2008)  have  reported  the  application  of  EID  to  industrial-scale 
or  large-scale  problems  in  the  petrochemical  and  nuclear  power  domains,  respectively. 

The  studies  listed  in  the  last  five  rows  of  Table  2  examined  the  application  of  EID  to  medium- 
scale  or  large-scale  problems  in  the  context  of  systems  other  than  process  control.  Positive 
results  have  been  obtained  against  all  three  criteria  for  information  retrieval  and  medical 
systems  as  well  as  network  management,  aviation,  and  military  command  and  control 
systems.  I  should  mention,  however,  that  there  are  only  one  or  two  cases  of  empirical 
evaluation  for  each  of  these  classes  of  systems.  Nevertheless,  collectively,  the  studies  listed  in 
Table  2  demonstrate  that  EID  can  be  applied  to  a  range  of  systems  and  that,  for  those  systems, 
EID  can  uncover  novel  information  requirements  that  lead  to  better  performance,  when 
compared  with  existing  interfaces. 

Three  sets  of  experimental  results  are  worth  discussing  in  more  detail.  First,  experimental 
studies  show  that  EID  generally  leads  to  better  performance  than  existing  interfaces  on  non¬ 
routine  or  complex  tasks,  with  no  disadvantage  on  routine  or  simple  tasks  (e.g.,  Duez  & 
Vicente,  2005;  Ham  &  Yoon,  2001b;  Lau  et  al.,  2008;  Pawlak  &  Vicente,  1996;  Xu  et  al.,  1999). 
This  finding  is  consistent  with  the  framework's  emphasis  on  supporting  workers  in  situations 
requiring  adaptive  problem-solving. 

Second,  experimental  studies  also  show  that  the  benefit  of  EID  is  due  to  its  information 
content,  not  just  its  visual  format  (Ham  &  Yoon,  2001b;  Xu  et  al.,  1999).  Specifically,  when  the 
visual  format  of  ecological  and  existing  interfaces  is  controlled,  such  that  both  present 
information  in  an  alphanumeric  format,  for  example,  EID  still  leads  to  better  performance. 

Third,  experimental  studies  have  demonstrated  that  higher  levels  of  abstraction  and  structural 
means-ends  relations  (i.e.,  links  between  information  at  different  levels  of  abstraction)  in  the 
abstraction-decomposition  space  play  a  key  role  in  the  improved  performance  obtained  with 
EID  displays  (e.g..  Burns,  2000;  Ham  &  Yoon,  2001a;  Ham,  Yoon  &  Han,  2008;  Janzen  & 
Vicente,  1998;  Vicente,  Christoffersen  &  Pereklita,  1995).  Specifically,  progressively  adding 
information  from  higher  levels  of  abstraction  to  an  EID  display  leads  to  increasingly  better 
levels  of  performance.  In  addition,  grouping  information  according  to  structural  means-ends 
relations  leads  to  better  performance  with  an  EID  display  compared  with  not  organising 
information  in  this  way. 
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In  summary,  these  findings  all  point  to  the  value  of  the  EID  framework  for  designing 
interfaces  to  support  worker  adaptation  in  a  variety  of  complex  sociotechnical  systems. 
Accordingly,  this  body  of  research  provides  experimental  evidence  for  the  validity  of  some  of 
the  concepts  of  cognitive  work  analysis  as  well  as  their  generalisability  to  systems  other  than 
those  that  led  to  their  development.  Thus  the  EID  program  has  gone  quite  some  way  in 
addressing  the  limitations  of  the  original  research  by  the  Riso  group. 

5.2  Challenges 

Several  challenges  remain  to  be  addressed  with  the  EID  framework,  as  identified  by  those 
working  in  this  area.  First,  further  empirical  evaluation  of  EID  is  still  necessary  for  a  range  of 
systems. 

Second,  issues  relating  to  sensor  unavailability,  noise,  and  failure  must  be  resolved  (e.g., 
Flajdukiewicz,  Vicente,  Doyle,  Milgram,  &  Burns,  2001;  Reising  &  Sanderson,  1998,  2002a, 
2002b,  2004;  Sharp  &  Helmicki,  1998;  St-Cyr,  2006;  St-Cyr  &  Vicente,  2004,  2005;  Vicente  & 
Rasmussen,  1992;  Vicente,  2002).  Specifically,  the  lack  of  availability  of  sensors  in  a  system  for 
obtaining  all  of  the  work  domain  information  necessary  for  an  EID  display  may  compromise 
performance  with  such  interfaces.  In  addition,  the  reliability  of  sensors  with  respect  to  noise 
and  failure  may  compromise  performance  with  EID  displays. 

Third,  extensions  to  the  EID  framework  may  be  necessary  or  beneficial  for  further  improving 
performance  with  such  displays.  Burns  (2000)  has  investigated  supplementing  EID  with  more 
specific  design  principles  such  as  those  relating  to  the  problem  of  visual  momentum  described 
by  Woods  (1984).  Watson  and  Sanderson  (2007)  have  proposed  extensions  to  EID  to  take 
advantage  of  the  auditory  modality  for  presenting  information  to  workers  instead  of  relying 
solely  on  visual  displays.  Jamieson,  Miller,  Ho,  and  Vicente  (2007)  and  Miller  and  Vicente 
(2001)  have  investigated  extending  EID  to  take  advantage  of  information  requirements 
generated  by  other  work  analysis  techniques,  such  as  hierarchical  task  analysis,  and  other 
dimensions  of  cognitive  work  analysis,  such  as  control  task  analysis. 

Fourth,  although  there  are  a  few  cases  of  the  adoption  of  EID  displays  by  industry,  significant 
technology  transfer  has  yet  to  be  achieved.  A  book  by  Burns  and  Hajdukiewicz  (2004)  on  EID, 
which  makes  this  framework  more  accessible  to  practitioners,  may  assist  with  facilitating  this 
transfer. 

Alongside  the  developments  in  EID,  two  other  seminal  monographs  on  cognitive  work 
analysis  were  published  in  the  1990s  by  Rasmussen  et  al.  (1994)  and  Vicente  (1999),  as 
indicated  in  Figure  3.  Arguably,  together  with  the  experimental  evidence  in  support  of  the 
EID  framework  and  thus  cognitive  work  analysis,  these  publications  were  contributory  factors 
to  the  next  major  phase  of  research  in  this  area,  which  extended  the  application  of  cognitive 
work  analysis  to  problems  other  than  interface  design. 
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6.  Beyond  Ecological  Interface  Design 

In  the  late  1990s  and  early  in  the  next  decade,  cognitive  work  analysis  was  applied  to  a  variety 
of  problems  including  the  definition  of  training  needs  and  training-system  requirements 
(Naikar  &  Sanderson,  1999),  development  of  software  specifications  (Leveson,  2000), 
evaluation  of  system  design  concepts  (Naikar  &  Sanderson,  2001),  development  of  a  team 
design  (Naikar,  Pearce,  Drumm  &  Sanderson,  2003),  definition  of  strategies  for  human  error 
management  (Naikar  &  Saunders,  2003),  and  development  of  recommendations  for 
automation  and  role  allocation  (Bisantz,  Roth,  Brickman,  Gosbee,  Hettinger  &  McKinney, 
2003).  Unlike  the  program  in  ecological  interface  design,  which  has  occurred  largely  within  a 
research  context,  all  of  these  applications  of  cognitive  work  analysis  occurred  in  industrial 
settings  or  within  a  professional  context.  In  what  follows,  I  discuss  the  application  of  cognitive 
work  analysis  to  one  of  these  problems  in  more  detail,  specifically,  team  or  organisational 
design. 

6.1  Team/Organisational  Design 

For  the  problem  of  team  or  organisational  design,  the  concepts  of  social  organisation  and 
cooperation  analysis  are  particularly  relevant  (Naikar,  2006,  in  press).  Like  those  of  the  other 
dimensions  of  cognitive  work  analysis,  the  concepts  of  this  dimension  are  informed  by  studies 
of  work  in  naturalistic  settings.  For  example,  the  research  of  Rochlin,  LaPorte,  and  Roberts 
(1987)  on  high  reliability  organisations  is  particularly  relevant  (Rasmussen  et  al.,  1994; 
Vicente,  1999). 

Rochlin  et  al.  (1987)  conducted  a  field  study  of  how  U.S.  Navy  personnel  on  aircraft  carriers  at 
sea  coordinate  their  work  activities.  They  found  that  the  formal  organisation  of  this  system, 
that  which  is  documented  on  paper,  is  rigid,  hierarchical,  and  centralised,  being  characterised 
by  clearly  defined  chains  of  command  and  means  to  enforce  authority.  Typically,  this 
organisational  structure  governs  operations  on  the  ship. 

During  complex  operations,  however,  a  very  different  type  of  organisational  structure  is 
adopted.  This  organisational  structure  may  be  described  as  informal,  given  that  it  is  not 
officially  documented.  The  informal  organisation  is  flat  and  distributed  rather  than 
hierarchical  and  centralised.  For  instance,  workers  at  lower  levels  of  the  hierarchy  have  the 
autonomy  to  make  critical  decisions  without  the  approval  of  those  higher  up.  The  informal 
organisation  is  also  flexible  in  that  there  is  no  pre-specified  plan  for  when  it  will  be  adopted. 
Furthermore,  the  specific  organisational  structure  that  is  adopted  on  any  one  occasion  is 
emergent,  which  in  this  case  means  that  there  is  no  simple  or  fixed  mapping  between  people 
and  roles  and,  therefore,  no  single  informal  organisational  structure.  Instead,  the  work 
organisation  on  the  ship  adapts  to  changes  in  circumstances.  According  to  Rochlin  et  al. 
(1987),  this  adaptability  contributes  greatly  to  achieving  the  balance  between  the  drive  for 
safety  and  reliability  and  the  drive  for  combat  effectiveness. 

So  how  are  the  concepts  of  cognitive  work  analysis  in  keeping  with  such  observations  of  work 
organisation  in  naturalistic  settings?  Whereas  Rasmussen  et  al.  (1994)  and  Vicente  (1999)  focus 
on  the  implications  of  these  observations  for  analysing,  and  thus  supporting,  existing 
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organisational  structures,  I  have  also  considered  the  implications  of  these  observations  for 
developing  new  team  or  organisational  designs  (Naikar,  2006,  in  press).  The  latter  issue  is 
what  I  emphasise  in  the  following  description  of  social  organisation  and  cooperation  analysis. 
Prior  to  doing  so,  I  should  mention  that  my  ideas  on  this  topic  have  not  been  formulated  fully 
or  tested  completely.  Therefore,  in  this  part  of  the  essay,  I  discuss  a  topic  on  which  my 
colleagues  and  I  are  currently  working. 

Cognitive  work  analysis  recognises  that,  in  complex  sociotechnical  systems,  flexible 
organisational  structures  that  can  be  adapted  to  local  contingencies  are  essential  for  dealing 
with  a  range  of  situations,  including  unanticipated  events.  Consequently,  there  is  no  attempt 
in  this  form  of  analysis  to  prescribe  a  single  or  best  organisational  structure.  Instead,  emphasis 
is  placed  on  defining  the  set  of  possibilities  for  work  organisation  in  a  system,  and  developing 
a  team  or  organisational  design  that  can  support  or  adopt  those  possibilities.  By  identifying 
the  set  of  possibilities  for  work  organisation  in  advance,  it  also  becomes  feasible  to  design 
interfaces,  training  systems,  and  other  aids  that  are  explicitly  tailored  to  those  possibilities,  so 
that  workers  are  provided  with  adequate  support  for  adopting  the  range  of  possibilities.  How, 
then,  does  one  define  the  set  of  possibilities  for  work  organisation  in  a  system? 

The  first  step  is  to  identify  the  criteria  that  may  shape  or  govern  how  work  is  allocated  or 
distributed  across  actors  in  a  range  of  situations.  Rasmussen  et  al.  (1994)  and  Vicente  (1999) 
describe  six  criteria.  Specifically,  these  are  the  competencies  of  actors,  the  access  that  actors 
have  to  information  or  the  means  for  action,  the  requirements  for  minimising  coordination  or 
communication,  the  necessity  for  workload  sharing,  the  requirements  for  safety  and 
reliability,  and  the  requirements  for  compliance  (with  regulations,  for  example).  Presumably, 
different  or  additional  criteria  may  be  applicable  depending  on  the  system  of  interest. 

Having  identified  the  criteria  that  are  relevant  to  a  system,  the  next  step  is  to  examine  how  the 
work  demands  of  the  system  may  be  allocated  or  distributed  across  actors  as  a  function  of 
applying  various  criteria;  the  criteria  may  be  applied  singly  or  in  different  combinations.  To 
perform  this  step,  representations  of  a  system's  work  demands  are  necessary.  The  work 
demands  of  a  system  may  be  modelled  in  terms  of  the  work  domain,  work  situations,  work 
functions,  control  tasks,  or  strategies  (Table  1).  Hence  the  modelling  tools  from  the  preceding 
dimensions  of  cognitive  work  analysis  are  relevant. 

Figure  8  illustrates  how  a  decision  ladder  model  of  an  uninhabited  aerial  vehicle  for  maritime 
surveillance  (Elix  &  Naikar,  2008)  may  be  used  to  examine  the  distribution  of  control  tasks 
across  actors.  Given  certain  criteria,  such  as  the  necessity  for  workload  sharing  or  the 
competencies  of  actors,  the  shaded  areas  of  the  decision  ladder  might  be  allocated  to  Actor  A 
and  the  remaining  areas  to  Actor  B.  This  representation,  then,  highlights  the  nature  of  the 
work  content  or  responsibilities  of  the  two  actors,  as  well  as  their  information,  coordination, 
and  resource  requirements,  given  that  work  content.  This  figure  represents  one  possibility  for 
work  organisation  in  the  system.  The  application  of  other  criteria  may  result  in  different 
possibilities  (e.g..  Figure  9). 
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required  destination  safely/efficientiy?  What  route(s) 
\  will  get  the  aircraft  to  the  required  destination 
safely/efficientiy?  To  which  destination  can  the 
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Figure  8  Use  of  the  decision  ladder  to  represent  one  possibility  for  work  organisation  in  an 
uninhabited  aerial  vehicle  for  maritime  surveillance 
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Figure  9  Use  of  the  decision  ladder  to  represmt  a  second  possibility  for  work  organisation  in  an 
uninhabited  aerial  vehicle  for  maritime  surveillance 


The  other  modelling  tools  of  cognitive  work  analysis  may  be  used  in  a  similar  fashion  to 
examine  the  possibilities  for  work  organisation  in  a  system.  To  offer  another  example.  Figure 
10  shows  how  an  information  flow  map  from  strategies  analysis  may  be  used  to  examine  the 
distribution  of  work  across  actors.  In  this  figure,  some  aspects  of  a  strategy  for  electronic 
troubleshooting  are  allocated  to  Actor  A  and  other  aspects  to  Actor  B.  (A  more  detailed 
description  of  this  strategy  is  provided  later  in  the  section  on  strategies  analysis.)  I  note  that 
although  I  have  used  a  single  decision  ladder  and  information  flow  map  for  illustration  in  this 
section,  for  most  complex  sociotechnical  systems,  multiple  decision  ladders  and  information 
flows  maps  are  likely  to  be  necessary  for  modelling  work  demands. 
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Figure  10  Use  of  an  information  flow  map  to  examine  the  possibilities  for  work  organisation  in  an 
electronic  troubleshooting  system.  Adapted  from  Rasmussen,  ].  (1981).  Models  of  mental 
strategies  in  process  plant  diagnosis.  In  ].  Rasmussen  &  W.  B.  Rouse  (Eds.),  Human 
detection  and  diagnosis  of  system  failures  (pp.  241-258).  ©  1981  Plenum  Press 
(Figure  4),  with  kind  permission  from  Springer  Science+Business  Media  B.V., 
incorporating  modifications  made  by  Vicente  (1999). 

Having  defined  the  set  of  possibilities  for  work  organisation  in  a  system,  the  next  step  is  to 
identify  the  team  or  organisational  design  requirements  necessary  for  supporting  each 
possibility2  (Figure  11).  For  example,  the  requirements  may  relate  to  the  team  size, 
hierarchical  structure,  subgroups,  or  roles  required  for  supporting  each  possibility.  Following 
that,  a  team  or  organisational  design  that  best  fulfils  the  set  of  requirements,  or  that  best 
supports  the  range  of  possibilities  for  work  organisation,  may  be  created. 


2  Requirements  relating  to  the  design  of  interfaces,  training  systems,  and  other  aids  for  supporting  the  possibilities 
may  also  be  produced.  Some  of  these  requirements  may  be  targeted  at  maximising  the  strengths  or  minimising  the 
limitations  of  each  possibility. 
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Set  of  possibilities  for  work  organisation 

I 

Team  or  organisational  design  requirements 
for  those  possibilities 

I 

Team  or  organisational  design 


Figure  11  General  process  for  creating  a  team  or  organisational  design  with  cognitive  work  analysis 

6.1.1  Application 

Previously,  we  have  used  these  concepts,  albeit  in  a  limited  way,  to  develop  a  team  design  for 
a  new  Airborne  Early  Warning  and  Control  system  or  AEW&C  (Naikar,  in  press;  Naikar  et  al., 
2003).  This  team  design,  which  was  adopted  by  the  Royal  Australian  Air  Force,  accommodates 
flexibility  in  the  size  of  the  team,  the  number  of  levels  of  hierarchy,  and  the  roles  or 
responsibilities  of  crew  members. 

Figure  12  shows  an  instantiation  of  the  AEW&C  team  design  in  which  the  role  of  deputy 
mission  commander  is  implemented  when  the  mission  commander  needs  to  be  buffered  from 
administrative  duties.  On  missions,  or  segments  of  missions,  when  this  buffer  is  not  required, 
the  person  in  this  role  can  assume  other  responsibilities  or  serve  as  a  spare  crew  member  to 
enable  the  rotation  of  crew  through  positions  and  thus  allow  more  or  longer  rest  periods  for 
members  of  the  team.  These  alternative  configurations  involve  variations  to  team  size, 
hierarchical  structure,  and  roles  or  responsibilities. 
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Figure  12  An  instantiation  of  the  team  design  for  AEW&C 

The  specific  process  used  to  develop  the  team  design  for  AEW&C  has  been  reported  in  detail 
previously  (Naikar,  in  press;  Naikar  et  al.,  2003).  The  approach  to  team  or  organisational 
design  described  in  this  essay  builds  on  this  process,  particularly,  by  addressing  its 
limitations.  The  details  of  this  newly  revised  approach,  as  well  as  its  strengths  and  limitations, 
are  currently  being  investigated.  Specifically,  we  are  using  this  approach  to  develop  team 
designs  for  operating  inhabited  and  uninhabited  vehicles  for  maritime  surveillance  (Elix  & 
Naikar,  2011).  In  addition,  we  are  using  this  approach  to  develop  an  organisational  design  for 
the  Royal  Australian  Air  Force  (Treadwell,  Yeung,  Brady  &  Naikar,  2011).  These  case  studies 
will  add  to  the  body  of  evidence  on  whether  cognitive  work  analysis  can  be  extended  to 
develop  new  team  or  organisational  designs  for  complex  sociotechnical  systems. 

6.1.2  Challenges 

One  of  the  challenges  we  face  in  applying  cognitive  work  analysis  to  problems  like  team  or 
organisational  design  in  industrial  settings  is  empirical  evaluation  (Naikar,  2009,  in  press).  It  is 
important  to  acknowledge  that  we  do  not  have  empirical  evidence  that  cognitive  work 
analysis  led  to  an  effective  team  design  for  AEW&C  or  to  a  better  team  design  for  this  system 
than  other  techniques  might  have.  It  is  difficult,  if  not  impossible,  to  conduct  such  studies  in 
industrial  settings  because  of  time  and  resource  constraints. 

To  illustrate,  in  order  to  establish  empirically  whether  cognitive  work  analysis  led  to  an 
effective  team  design  for  AEW&C,  access  to  the  real  aircraft  or  mockups,  prototypes,  or 
simulations  of  relevant  aspects  of  the  aircraft  are  necessary.  Such  resources  were  not  available 
when  the  team  design  for  AEW&C  was  developed.  At  that  time,  AEW&C  was  a  future 
system,  which  the  Australian  Government  intended  to  acquire.  It  only  existed  as  a  concept  -  it 
had  not  been  built  or  manufactured  anywhere  in  the  world.  Thus  there  were  no  real  aircraft, 
mockups,  prototypes,  or  simulations.  Another  reason  that  empirical  evaluation  was  not  viable 
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at  that  time  is  that  AEW&C  is  a  first-of-a-kind  system.  It  incorporates  significant  advances  in 
technology  compared  with  what  were  then  current-generation  systems.  Hence  it  was  not 
possible  to  use  existing  systems  to  evaluate  the  team  design  for  AEW&C. 

When  mockups,  prototypes,  or  simulations  of  the  aircraft  become  available,  or  when  the  real 
aircraft  are  brought  into  operation,  there  are  other  challenges  with  establishing  whether 
cognitive  work  analysis  led  to  an  effective  team  design  for  AEW&C.  The  main  difficulty  is 
distinguishing  the  contribution  of  cognitive  work  analysis  to  the  performance  of  the  team 
design  from  other  aspects  of  the  aircraft's  design,  such  as  its  interface  design.  For  instance,  if 
the  team  design  performs  well,  it  may  be  the  case  that  the  performance  of  the  team  is  due  to 
the  interface  design  and  that  any  team  design  would  have  performed  well  with  that  interface. 

Similar  challenges  are  faced  in  establishing  whether  cognitive  work  analysis  led  to  a  better 
team  design  for  AEW&C  than  other  techniques.  Specifically,  for  this  to  be  determined,  it 
would  be  necessary  to  employ  multiple  techniques  for  team  design  and  to  compare  the 
performance  of  the  resulting  designs  using  the  real  aircraft  or  suitable  mockups,  prototypes, 
or  simulations.  The  time  and  resources  necessary  for  such  a  study  were  not  available  on  this 
project. 

6.1.3  Evaluation 

Such  challenges  with  the  empirical  validation  of  techniques  in  industrial  settings  are  not 
limited  to  cognitive  work  analysis.  The  challenges  also  apply  to  other  techniques  employed  in 
industry,  which  is  why  these  techniques  are  rarely  evaluated  formally.  Instead,  the  techniques 
are  assessed  principally  against  the  criterion  of  usefulness  (Czaja,  1997;  Whitefield,  Wilson  & 
Dowell,  1991;  Vicente,  1999).  Accordingly,  I  have  evaluated  applications  of  cognitive  work 
analysis  for  the  Australian  Department  of  Defence  against  two  measures  of  usefulness 
(Naikar,  2009,  in  press).  The  first  is  impact  or  the  ability  to  influence  practice  on  a  project.  The 
second  is  uniqueness  or  the  ability  to  make  a  unique  contribution  to  a  project  compared  with 
techniques  commonly  used  in  industry.  How,  then,  does  the  application  of  cognitive  work 
analysis  to  design  a  team  for  AEW&C  fare  against  these  criteria? 

There  were  two  main  demonstrations  that  cognitive  work  analysis  had  impact  or  influenced 
practice  on  this  project.  First,  the  team  design  that  was  developed  for  AEW&C  was  adopted 
by  the  Australian  Defence  Department.  This  team  design  was  judged  by  subject  matter 
experts  to  be  suitable  for  AEW&C  and  better  than  other  team  designs  they  had  considered.  As 
it  was  not  feasible  to  perform  an  empirical  evaluation  of  this  team  design,  the  Defence 
Department  implemented  a  number  of  strategies  to  manage  the  risk  that  it  may  be  unsuitable 
for  AEW&C.  Second,  on  the  basis  of  our  analyses,  the  technical  specifications  for  the  aircraft 
were  modified  so  that  it  better  supported  the  team  design  for  AEW&C.  As  this  was  done  early 
in  the  acquisition  cycle,  prior  to  the  signing  of  a  contract  with  the  aircraft  manufacturer,  the 
alterations  were  achieved  without  cost  to  the  Australian  Government. 

Cognitive  work  analysis  also  made  a  unique  contribution  to  this  project  because  standard 
techniques  for  team  design  cannot  readily  be  applied  to  future,  first-of-a-kind  systems.  The 
standard  techniques,  which  span  a  variety  of  disciplines  including  engineering,  social  and 
organisational  psychology,  and  engineering  psychology,  differ  markedly  in  their  theoretical 
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orientations  to  team  design  (Davis  &  Wacker,  1982, 1987;  Hackman  &  Oldham,  1980;  Lehner, 
1991;  Medsker  &  Campion,  1997;  Sundstrom,  De  Meuse  &  Futrell,  1990).  Nevertheless,  all  of 
the  techniques  depend  on  a  work  analysis  of  the  system  of  interest. 

Some  of  the  techniques  rely  on  studying  or  observing  how  work  is  done  in  a  system  in  order 
to  produce  a  team  design  for  it.  However,  as  future,  first-of-a-kind  systems  are  not 
functioning  systems,  and  have  no  equivalent  existing  systems,  the  way  in  which  work  is  done 
in  these  systems  cannot  be  studied  or  observed.  Other  techniques  rely  on  specifying  how 
work  should  be  done  in  a  system  in  terms  of  a  set  of  task  sequences  or  procedures  in  order  to 
create  a  team  design  for  it.  However,  task  sequences  or  procedures  cannot  be  specified  in  full 
for  future,  first-of-a-kind  systems  because  the  details  of  their  technical  solutions  are  still 
largely  undefined,  workers  often  develop  novel  work  practices  as  they  gain  experience  with  a 
new  system,  and  workers  may  have  to  deal  with  unanticipated  events,  for  which  task 
sequences  or  procedures  cannot  be  specified  in  advance. 

In  contrast  to  standard  techniques  for  team  design,  cognitive  work  analysis  is  especially  well 
suited  to  designing  teams  for  future,  first-of-a-kind  systems.  The  analysis  of  the  constraints  on 
actors'  behaviour  does  not  require  a  functioning  system.  Moreover,  these  constraints  can 
accommodate  many  different  trajectories  of  behaviour,  including  task  sequences  or 
procedures  that  are  difficult  to  specify  a  priori.  Overall,  then,  as  judged  against  the  criteria  of 
impact  and  uniqueness,  cognitive  work  analysis  appears  to  provide  a  useful  approach  for 
designing  teams  for  future,  first-of-a-kind  systems. 

Finally,  it  is  worth  considering  briefly  how  applications  of  cognitive  work  analysis  to 
problems  other  than  interface  design  fare  against  the  criteria  outlined  by  Vicente  (2002)  for 
assessing  the  EID  framework.  Table  3  summarises  the  results  of  four  projects  conducted  for 
the  Australian  Department  of  Defence.  Without  delving  into  the  details  of  these  projects,  this 
table  shows  that  the  proof  of  principle  and  analytical  criteria  were  satisfied  in  every  case.  (For 
comprehensive  descriptions  of  these  projects,  see  Naikar  (2009,  in  press)  or  the  original 
publications  cited  in  Table  3.)  That  is,  these  projects  demonstrated  that  cognitive  work 
analysis  can  be  applied  to  evaluate  system  design  concepts,  develop  team  designs,  define 
training  needs  and  training-system  requirements,  and  define  strategies  for  managing  human 
error.  Moreover,  for  each  application,  cognitive  work  analysis  led  to  unique  results  compared 
with  existing  solutions  or  the  results  of  other  techniques.  The  empirical  criterion,  however, 
was  not  satisfied  on  any  project  due  to  time  and  resource  constraints.  A  shift  may  be  necessary 
from  an  industrial  setting  to  a  research  setting  for  such  studies  to  be  conducted.  Further  case 
studies  of  these  applications  of  cognitive  work  analysis  across  a  range  of  systems,  beyond  just 
military  ones,  are  also  necessary. 
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Table  3  Assessment  of  four  applications  of  cognitive  work  analysis  to  problems  other  than  interface 

design  against  proof-of-principle  (P-of-P),  analytical,  and  empirical  criteria 


Application 

Publication 

P-of-P: 

Applied? 

Analytical: 
New  Results? 

Empirical: 

Perform? 

Evaluation  of 
system  design 
concepts 

Naikar  &  Sanderson, 
1999 

Y 

Y 

Team  design 

Naikar  et  al.,  2003 

Y 

Y 

Training  needs 
and  training- 
system 
requirements 

Naikar  &  Sanderson, 
1999 

Y 

Y 

Human  error 
management 

Naikar  &  Saunders, 
2003 

Y 

Y 

7.  Methodology 

Together  with  the  results  of  the  experimental  program  on  EID,  the  outcomes  of  novel 
applications  of  cognitive  work  analysis  in  industrial  settings  are  heartening.  They  suggest, 
among  other  things,  that  the  more  extensive  use  of  cognitive  work  analysis  for  the  analysis, 
design,  and  evaluation  of  work  in  sociotechnical  systems  may  be  fruitful.  The  widespread 
adoption  of  cognitive  work  analysis  for  this  purpose,  though,  is  hampered  by  the  fact  that  this 
approach  is  challenging,  not  only  to  learn,  but  also  to  perform  well.  While  the  texts  by 
Rasmussen  et  al.  (1994)  and  Vicente  (1999)  have  made  cognitive  work  analysis  much  more 
accessible  to  analysts  than  it  was  previously,  both  texts  focus  on  providing  a  conceptual 
perspective  of  this  framework.  As  highlighted  by  Vicente  (1999),  this  scope  was  a  deliberate 
choice  because  cognitive  work  analysis  was  still  relatively  unknown  at  the  time.  Now, 
however,  it  is  clear  that  the  accessibility  of  cognitive  work  analysis  may  be  improved  further 
with  the  development  of  a  complementary  methodological  perspective. 

As  shown  in  Figure  3,  the  development  of  a  methodology  for  cognitive  work  analysis  was  the 
next  major  phase  of  research  in  this  area.  Although  my  colleagues  and  I  have  been  conducting 
methodological  research  on  several  aspects  of  cognitive  work  analysis,  in  this  essay  I  focus  on 
my  most  recent  research  on  work  domain  analysis  (Naikar,  in  press).  While  the  details  of  the 
methodology  I  have  developed  for  work  domain  analysis  are  beyond  the  scope  of  this  essay,  I 
explain  some  of  the  challenges  I  encountered  in  its  development,  which  conveys  its  scope. 

7.1  Terminology 

One  challenge  I  faced  was  terminology.  The  terms  used  for  naming  and  describing  concepts, 
and  whether  these  terms  are  defined  unambiguously  and  employed  consistently,  have  a 
significant  effect  on  one's  understanding  of  what  work  domain  analysis  is,  and  thus  how  it 
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should  be  performed.  Is  there  a  difference  between  a  system  and  a  work  domain  for  example? 
Can  these  terms  be  used  interchangeably  and,  if  not,  what  is  the  difference  between  them?  In 
his  book  on  cognitive  work  analysis,  Vicente  (1999)  provides  a  reasonably  comprehensive 
glossary.  In  developing  a  methodology  for  work  domain  analysis,  my  aim  was  to  use  terms  in 
a  way  that  was  as  consistent  with  this  glossary  as  possible.  There  were  instances,  however, 
when  I  thought  it  justifiable  to  depart  from  the  definitions  provided  in  Vicente's  glossary. 

One  example  is  the  usage  of  the  term  system.  This  term  is  defined  by  Vicente  (1999)  as  "A  set 
of  interrelated  elements  that  share  a  common  Goal  or  Purpose"  (p.  9).  He  states  that  a  system 
can  be  a  sociotechnical  system,  work  domain,  or  actor,  whereby  an  actor  is  a  worker  or 
automation.  This  can  be  taken  to  mean  that  the  term  system  may  be  used  to  refer  to  four  very 
different  concepts.  Therefore,  unless  it  is  somehow  made  clear  how  the  term  system  is  being 
used  whenever  it  is  mentioned,  there  is  much  scope  for  creating  misunderstanding  of  the 
concepts  and  methodology  of  cognitive  work  analysis. 

For  this  reason,  in  discussing  methodology  for  work  domain  analysis  (Naikar,  in  press),  I  have 
elected  to  use  the  term  system  only  when  referring  to  a  sociotechnical  system,  which  Vicente 
(1999)  defines  as  "A  System  composed  of  technical,  psychological,  and  social  elements"  (p.  9). 
Admittedly,  this  is  a  somewhat  arbitrary  choice,  but  the  bottom  line  is  that  appreciating  the 
distinctions  between  the  four  concepts  listed  above  is  critical  for  understanding  and 
performing  work  domain  analysis.  Using  the  term  system  to  refer  to  all  four  of  these  concepts, 
or  more  than  one  concept,  obscures  these  critical  distinctions.  In  particular,  a  work  domain  is 
not  the  same  thing  as  a  sociotechnical  system,  but  there  are  many  instances  when  the  term 
system  is  used  to  refer  to  both  of  these  concepts  and  sometimes  it  is  unclear  which  concept  is 
under  discussion. 

Arguably,  the  most  significant  departure  I  have  made  from  Vicente's  (1999)  glossary  is  the 
definition  of  the  term  work  domain,  not  because  Vicente's  definition  is  incorrect,  but  because  it 
can  be  misinterpreted.  Vicente's  definition  is  also  limited  in  some  respects  given  the  range  of 
systems  and  problems  to  which  cognitive  work  analysis  is  now  being  applied.  Vicente  (1999) 
defines  a  work  domain  as  "The  System  being  controlled,  independent  of  any  particular  Worker, 
Automation,  Event,  Task,  Goal,  or  Interface"  (p.lO).  This  definition  has  been  taken  to  mean, 
among  other  things,  that  a  work  domain  model  cannot  include  control  systems  such  as 
automation  (Miller,  2004;  Miller  &  Sanderson,  2000).  It  is  not  clear  to  me  that  this  is  what 
Vicente  intended.  My  interpretation  of  his  work  is  that  work  domain  models  can  include  the 
structural  properties,  or  relatively  permanent  properties,  of  control  systems,  but  not  their 
behaviour,  which  is  consistent  with  the  theoretical  underpinnings  of  work  domain  analysis 
(Naikar,  in  press). 

Nevertheless,  because  the  issue  of  the  representation  of  control  systems  in  a  work  domain 
model  is  important  for  many  systems  and  applications,  and  given  that  Vicente's  (1999) 
definition  has  caused  confusion  on  this  and  other  similar  points,  I  have  defined  the  term  work 
domain  instead  as  the  functional  structure  of  the  physical,  social,  or  cultural  environment  of 
actors  in  a  system,  which  places  constraints  on  their  behaviour  (Naikar,  in  press).  Following 
Vicente,  I  use  the  term  functional  to  mean  "action-relevant"  (p.155)  and  the  term  structure  to 
mean  "A  relatively  permanent  relational  property  of  a  System"  (p.9). 
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My  definition  of  work  domain  differs  from  Vicente's  (1999)  definition  in  emphasis  more  than  in 
semantic  content.  The  term  functional  in  my  definition  is  similar  to  the  concept  of  the  system 
being  controlled  in  Vicente's  definition.  The  main  difference  is  that  the  concept  of  relevance 
for  action  can  be  more  readily  extended  to  a  broader  range  of  systems,  particularly  those  in 
which  the  constraints  on  actors'  behaviour  originate  in  human  intentions  rather  than  in 
physical  or  natural  laws.  The  term  structure,  which  in  my  definition  refers  to  a  relatively 
permanent  property  of  a  system,  is  similar  to  Vicente's  concept  of  independence  from  any 
particular  worker,  automation,  event,  task,  goal  or  interface.  However,  my  definition  makes 
clearer  that  a  work  domain  model  can  include  the  structural  properties  of  objects  in  the 
environment  of  actors  such  as  automation,  for  example.  The  term  environmen  t  in  my  definition 
is  consistent  with  Vicente's  (1999)  and  Rasmussen  et  al.'s  (1994)  descriptions  of  the  work 
domain  as  representing  the  environment,  field,  landscape,  or  territory  in  which  workers 
operate.  Such  terms  are  more  naturally  associated  with  physical  surroundings  or  conditions, 
but  workers  in  sociotechnical  systems  operate  within  a  social  or  cultural  environment  as  well. 
Work  domain  analysis,  then,  is  concerned  with  analysing  the  functional  structure  of  the 
physical,  social,  or  cultural  environment  of  actors  in  a  system,  as  all  of  these  types  of 
environments  can  place  constraints  on  their  behaviour. 

7.2  Guidelines 

A  second  challenge  I  faced  was  formulating  guidelines  for  work  domain  analysis  in  a  form 
that  remains  faithful  to  how  this  type  of  analysis  is  actually  performed  by  experienced 
analysts,  but  still  serves  as  an  instructional  tool  for  novices.  Initially,  the  guidelines  were 
presented  as  a  series  of  steps  (Naikar,  Hopcroft  &  Moylan,  2005).  However,  as  the  process  of 
performing  work  domain  analysis  is  flexible  and  iterative,  not  fixed  and  linear,  I  reformulated 
the  guidelines  as  a  set  of  analytic  themes  or  questions  for  analysts  to  consider  when 
performing  work  domain  analysis  (Naikar,  in  press).  These  analytic  themes  (Figure  13),  which 
are  intertwined  and  difficult  to  separate  in  practice,  represent  the  major  concerns  that  analysts 
confront  recurrently  while  performing  a  work  domain  analysis.  Analysts  may  consider  the 
analytic  themes  in  any  order  and  several  analytic  themes  may  be  under  deliberation  at  any 
one  time.  Thus  the  analytic  themes  may  be  visualised  as  bubbles  that  appear  and  disappear 
continually,  in  any  order,  either  singly  or  several  at  a  time. 
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Theme  6: 
What  are 
the  sources 
of  information 
for  the 
analysis? 


Theme  8: 

Is  the 

abstraction- 
decomposition 
space  a  valid 
model  of  the 
focus  system? 


Theme  2: 
What  are 
the  project 
restrictions? 


Theme  3: 
What  are  the 
boundaries  of  the 
analysis  or  what 
is  the  focus 
system  for  the 
analysis? 


Theme  1: 
What  is  the 
purpose  of 
the  analysis? 


Theme  7: 
What  is  the 
content  of  the 
abstraction- 
decomposition 
space? 


Theme  5: 
Where  on  the 
causal- 
intentional 
continuum 
does  the  focus 
system  fall? 


Figure  13  Analytic  themes  for  work  domain  analysis 


7.3  Multiple  Models 

In  developing  the  guidelines,  I  avoided  providing  a  personal  account  of  how  I  perform  work 
domain  analysis.  Instead,  I  reviewed  hundreds  of  publications,  produced  by  many  different 
analysts,  to  establish  the  most  powerful  and  useful  ways  of  creating  work  domain  models. 
This  point  may  be  illustrated  in  relation  to  the  analytic  theme  Is  it  Useful  to  Develop  Multiple 
Models? 

One  reason  that  analysts  create  multiple  work  domain  models  of  a  system  is  to  represent 
different  stakeholders'  perspectives  of  a  problem  (Naikar,  in  press).  Stakeholders  may  be 
described  as  individuals  or  groups  with  different  views  of  a  particular  problem.  The  views  or 
perspectives  of  stakeholders  may  be  described  as  their  object  worlds.  Stakeholders  generally 
have  something  to  gain  or  lose  with  respect  to  the  problem  under  consideration,  which  means 
that  they  usually  have  to  interact  or  coordinate  their  activities  to  achieve  their  respective 
objectives.  By  modelling  the  perspectives  of  multiple  stakeholders,  it  becomes  possible  to 
introduce  designs  for  one  stakeholder  without  creating  unintended  side  effects  on  other 
stakeholders.  In  addition,  designs  may  be  produced  that  allow  stakeholders  to  coordinate 
their  activities  more  effectively. 
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One  example  of  multiple  models  of  stakeholders'  perspectives  was  developed  by  Burns  and 
Vicente  (2000)  for  an  engineering  design  system  (Figure  14).  The  stakeholders  in  this  system 
were  all  concerned  with  the  design  of  a  control  panel  for  the  control  room  of  a  nuclear  power 
plant.  The  stakeholders  included  human  factors  designers,  structural  engineers, 
implementers,  the  customer,  and  management.  Each  stakeholder  had  a  different  view  or 
perspective  of  the  design  of  the  control  panel.  For  instance,  whereas  human  factors  engineers 
were  concerned  with  the  comfort  and  usability  of  the  design,  structural  engineers  were 
concerned  with  designing  a  control  panel  with  a  certain  strength  and  rigidity.  This 
representation  of  multiple  stakeholders'  perspectives  is  typical  of  those  produced  by  most 
researchers  in  that  there  is  one  work  domain  model  for  each  stakeholder. 


HF  Design 

Structural 

Design 

Implementers 

Customer 

(Utility) 

Upper 

Management 

View 

display  surface 
for  indicators 
and  controls 

physical  housing 
for  indicators 
and  controls 

something 
they  have  to 
produce 

furnishings 
in  their 
control  room 

contract 

completion 

Objectives 

visibility, 

operability 

strength, 

stability 

feasibility 

image,  cost 

marketshare, 
on  time, 
within  budget 

Processes 

viewing  angles, 
reach  envelopes 

ft 

seismic  testing  / 

J8> 

manufacturing 
processes,  shipping 
installation,  on-site 
modifications^)*^ 

approval 
,  process 
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resource 

allocation 

Physical 

Components 

(  panel  dimensions,  panel  geometry,  room  configuration  ^ 

schedule, 
personnel,  $, 
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anthropometric\ 

(  constructi 

y' 

on  materials  ) 

room  dimensions, 
building  dimensions 
plant  stating 

Figure  14  Multiple  models  of  stakeholders  in  an  engineering  design  system.  From 
Burns,  C.  M.,  &  Vicente,  K.  ].,  A  framework  for  describing  and 

understanding  interdisciplinary  interactions  in  design,  Proceedings  of  DIS  '95: 
Symposium  on  Designing  Interactive  Systems,  97-103,  ©  ACM,  Inc. 
http://doi.acm.org/10.1145/225434.225445.  Reprinted  by  permission,  incorporating 
modifications  made  by  Vicente  (1999). 


There  is,  however,  one  exception  and  that  is  a  representation  of  multiple  stakeholders' 
perspectives  produced  by  Rasmussen,  Pejtersen  and  Schmidt  (1990)  and  Rasmussen  et  al. 
(1994)  for  the  treatment  of  patients  in  a  hospital  (Table  4  and  Figure  15).  The  important  point 
to  note  about  Table  4  is  that  none  of  the  labels  in  the  top  row  signify  stakeholders.  Instead,  the 
labels  signify  work  domains,  independently  of  any  stakeholders,  that  must  be  considered  in 
the  treatment  of  patients  in  a  hospital.  These  work  domains  include  a  patient's  private  life  and 
health  as  well  as  a  hospital's  cure,  care,  and  administrative  fields  of  interest. 
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Table  4  Work  domains  that  must  be  considered  in  the  treatment  of  patients  in  a  hospital.  Adapted 
from  Rasmussen,  ].,  Pejtersen,  A.  M.,  &  Goodstein,  L.  P.  Cognitive  systems 
engineering.  Copyright  1994  John  Wiley  &  Sons.  Reproduced  with  permission  of  John 
Wiley  &  Sons,  Inc. 


PATIENT 

HOSPITAL 

Private  Life 

Health 

Cure 

Care 

Administration 

Goals  and 
Constraints 

Working  relations 
and  conditions; 
Family  relations; 
Goals  and 
constraints  of  plans 
and  commitments 

Effects  of  illness 
and  treatment  on 
person's  ability  to 
meet  subjective 
goals  and  criteria 

Cure  patient; 
Research,  Training 
MDs;  Public 
opinion;  Legal, 
economic,  and 
ethical  constraints 

Patient  well  being, 
physical  and 
psychic  care; 

Public  opinion, 
economic  and  legal 
constraints 

Laws  and 
regulations  of 
society,  associa¬ 
tions  and  unions; 
Workers'  protection 
regulations  etc. 

Priority 

Measures, 

Flow  of 

Values  and 

Material 

Personal  economy, 
Probability  of 
unemployment, 
cure,  etc. 

Probability  of  cure, 
priority  measures, 
pace  versus 
side-effects,  etc. 

Categories  of 
diseases:  Cost  of 
treatments,  patient 
suffering,  research 
relevance 

Flow  of  patients 
according  to 
category; 

Treatment,  and 
load  on  staff  and 
facilities 

Distribution  of 
funds  on  activities; 
Flow  of  material 
and  personnel 
to  diseases, 
departments 

General 

Functions 

and 

Activities 

Work  functions; 
Family  relations; 
Living  conditions 

State  of  health; 
Diseases  and 
possible  treatments 

Cure,  diagnostics, 
surgery, 
medication,  etc. 
Research,  clinical, 
experiments 

Board  and  lodging; 
Hygiene;  Social 
Care,  Physical 
support, 

transportation,  etc. 

Personnel  and 
material 
administration, 
Accounting,  sales 
and  purchase 

Physical 

Activities 
in  Work, 

Physical 

Processes  of 
Equipment 

Physical  work 
activities,  spare 
time  and  sports 
activities; 

Homework; 
Transportation,  etc. 

Specific  organic 
disorders  and 
possible  treatment, 
Previous  illness 
and  cures 

Specific  research 
and  treatment 
procedures;  Use 
of  tools  and 
equipment 

Monitoring, 
treating,  moving, 
cleaning  and 
serving  patients; 
Psychic  Care 

Processes  in  the 
administrative 
functions.  Office 
and  planning 
procedures 

Appearance, 
Location  and 
Configuration  of 
Material  Objects 

Patient 

identification,  age, 
address, 
profession, 
education,  family 
members,  etc. 

Physical  state  of 
patient,  weight, 
height,  previous 
treatments,  etc. 

Material  resources, 

patients, 

personnel, 

equipment; 

Medicine,  tools, 

etc. 

Facilities  and 
equipment  in 
patient  quarters, 
kitchens,  etc. 
Inventory  of  linen, 
food,  etc. 
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Figure  15  maps  the  stakeholders  who  are  concerned  with  the  treatment  of  patients  in  a 
hospital  onto  the  various  work  domain  models.  The  stakeholders  include  a  patient's  general 
medical  practitioner  as  well  as  a  hospital's  medical  doctors,  general  manager,  head  nurses, 
nurses,  and  assistant  nurses.  This  representation  shows  that  a  stakeholder  may  be  concerned 
with  more  than  one  work  domain.  Nurses,  for  example,  are  concerned  with  the  work  domains 
of  cure  and  care  as  well  as  administration.  In  addition,  several  stakeholders  may  be  concerned 
with  the  same  work  domain,  although  with  different  but  overlapping  aspects.  For  example,  all 
of  the  stakeholders  on  the  right  of  Figure  15  are  concerned  with  the  work  domain  of  care  but 
they  have  different,  yet  overlapping,  spheres  of  concern. 
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Figure  15  Stakeholders  concerned  with  the  treatment  of  patients  in  a  hospital  mapped  onto  the  work 
domains  in  Table  1.  Adapted  from  Rasmussen,  Pejtersen  &  Schmidt  (1990),  with 
permission. 


Such  insights  are  possible  because  Rasmussen  et  al.'s  (1990, 1994)  representation  of  the  health 
care  system  differentiates  between  work  domains  and  object  worlds  or,  in  other  words, 
between  work  domains  and  stakeholders'  views  of  those  work  domains.  Although  Burns  and 
Vicente  (2000)  observed  similar  interactions  between  stakeholders,  object  worlds,  and  work 
domains  in  their  study,  their  representation  of  the  engineering  design  system  could  not  depict 
those  interactions  because  their  work  domain  models  were  not  independent  of  stakeholders 
and  their  object  worlds.  Thus  the  representation  of  the  health  care  system  is  a  more  powerful 
model  because  it  can  accommodate  observations  of  how  work  is  achieved  in  that  system3. 

It  is  also  important  to  point  out  that  the  representation  of  the  health  care  system  in  Figure  15  is 
consistent  with  the  theoretical  underpinnings  of  work  domain  analysis.  Whereas  work 
domains  are  event-independent,  stakeholders  and  their  object  worlds  may  vary  as  a  function 


3 1  should  note  that  it  is  not  my  intention  to  single  out  Bums  and  Vicente  (2000)  for  criticism.  As  I  indicated 
earlier,  except  for  Rasmussen  et  al.  (1990, 1994),  all  other  researchers  who  have  modelled  multiple  stakeholders’ 
perspectives  have  produced  the  same  type  of  representation  as  Burns  and  Vicente  have  done. 
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of  the  situation,  or  over  time,  as  indicated  during  the  earlier  discussions  on  team  or 
organisational  design.  For  example,  the  roles  of  head  nurses  and  assistant  nurses,  and  thus 
their  object  worlds,  may  shift  or  adapt  in  response  to  local  contingencies,  while  the  constraints 
associated  with  the  work  domains  of  cure,  care,  and  administration  in  the  health  care  system 
remain  relatively  constant.  Hence  maintaining  the  distinction  between  work  domains,  which 
are  event-independent,  and  stakeholders  and  their  object  worlds,  which  are  not,  is  important. 

The  challenge  I  grappled  with,  then,  was  how  to  formulate  the  guidelines  for  work  domain 
analysis  so  that  representations  produced  are  consistent  with  that  of  the  health  care  system, 
rather  than  the  engineering  design  system,  when  modelling  multiple  stakeholders' 
perspectives.  Without  delving  into  the  details  of  the  guidelines,  the  answer  lies  in  how  the 
boundaries  for  a  work  domain  analysis  are  formulated.  That  is,  the  boundaries  must  be 
formulated  in  a  way  that  is  independent  of  any  specific  actors,  even  when  one  wants  to  model 
the  perspectives  of  particular  stakeholders  (Naikar,  in  press). 

7.4  Further  Methodological  Developments 

There  are  some  other  methodological  developments  in  cognitive  work  analysis,  which  I 
cannot  describe  in  detail  here.  For  control  task  analysis,  these  include  a  new  modelling  tool 
called  the  contextual  activity  template  (Naikar,  Moylan  &  Pearce,  2006),  a  strategy  for  building 
decision  ladder  models  (Elix  &  Naikar,  2008),  and  knowledge-elicitation  techniques 
(Lamoureux  &  Chalmers,  2009).  For  worker  competencies  analysis,  the  main  methodological 
development  is  a  skills,  rules,  and  knowledge  inventory  (Kilgore,  St-Cyr  &  Jamieson,  2009). 

8.  Beyond  Work  Domain  Analysis 

The  final  stage  of  research  in  cognitive  work  analysis  that  this  essay  addresses  is  the  recent 
emphasis  on  the  dimensions  of  the  framework  subsequent  to  work  domain  analysis  (Figure  3). 
These  dimensions  have  not  been  neglected  completely,  as  is  evident  from  the  preceding 
discussion.  However,  the  majority  of  work,  by  far,  has  focused  on  work  domain  analysis. 

The  initial  work  on  the  other  dimensions  has  focused  on  exemplifying  their  concepts  and 
investigating  their  applications  (e.g.,  Bisantz  &  Burns,  2009;  Jenkins,  Stanton,  Salmon  & 
Walker,  2009;  Lintern,  2009;  Naikar,  2006).  The  book  edited  by  Bisantz  and  Burns  also  contains 
chapters  that  offer  a  preliminary  exploration  of  the  form  the  subsequent  dimensions  might 
take  beyond  how  they  are  currently  understood.  In  the  following  discussion,  I  highlight  some 
of  the  challenges  associated  with  the  concepts  and  methodology  of  the  subsequent  dimensions 
using  strategies  analysis  as  an  example.  I  believe  that  the  work  of  the  naturalistic  decision 
making  community  is  relevant  for  addressing  these  challenges,  even  more  so  since  it  has 
expanded  its  program  to  encompass  functions  other  than  decision  making. 

8.1  Strategies  Analysis 

Strategies  analysis  has  several  distinctive  characteristics  (Naikar,  2006,  2010,  in  press; 
Rasmussen  et  al.,  1994;  Vicente,  1999).  First,  this  dimension  of  analysis  is  not  concerned  with 
specifying  detailed  sequences  of  actions  or  mental  operations.  Instead,  it  is  concerned  with 
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defining  categories  of  cognitive  procedures,  which  are  idealised,  abstract  descriptions  of 
particular  processes. 

Second,  strategies  analysis  recognises  that  several  strategies  are  usually  possible  for 
performing  a  single  activity.  This  dimension  of  analysis,  therefore,  involves  examining  the 
range  of  strategies  that  are  possible  for  particular  activities.  Pejtersen  (1979),  for  example, 
established  that  library  staff  and  users  could  adopt  many  different  strategies  for  finding 
material  or  information  that  satisfy  their  needs,  namely,  search  by  analogy,  bibliographical 
strategy,  analytical  search,  empirical  search,  and  browsing  strategy. 

Similarly,  Rasmussen  and  Jensen  (1973)  found  that  electronic  technicians  could  adopt  many 
different  strategies  for  diagnosing  faults  in  computer  equipment.  For  instance,  technicians 
could  adopt  a  pattern  recognition  strategy  (Figure  16).  That  is,  a  technician  may  recognise  a 
pattern  of  data  from  the  failed  equipment  as  being  familiar  and  associate  that  pattern  with  the 
cause  of  the  fault  or  a  task  to  perform,  for  example.  Alternatively,  technicians  could  adopt  a 
hypothesis-and-test  strategy  (Figure  17).  This  is  a  more  demanding  cognitive  strategy 
whereby,  stated  briefly,  a  hypothesis  about  the  failed  equipment  is  applied  to  a  model  of  the 
normal  equipment  to  produce  a  model  of  the  hypothetical  failed  condition.  The  model  of  the 
hypothetical  failed  condition  is  then  compared  with  observations  of  the  failed  equipment.  If 
there  is  a  match,  the  hypothesis  is  accepted.  If  not,  it  is  rejected,  and  either  another  hypothesis 
is  tested  or  an  alternative  strategy  is  adopted. 


Figure  16  Pattern  recognition  strategy  for  electronic  troubleshooting.  Adapted  from  Rasmussen,  ]. 

(1981).  Models  of  mental  strategies  in  process  plant  diagnosis.  In  ].  Rasmussen  &  E.  B. 
Rouse  (Eds.),  Human  detection  and  diagnosis  of  system  failures  (pp.  241-258).  © 
1981  Plenum  Press  (Figure  4),  with  kind  permission  from  Springer  Science+Business 
Media  B.V. 
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Figure  17  Hypothesis-and-test  strategy  for  electronic  troubleshooting.  Adapted  from  Rasmussen,  ]. 

(1981).  Models  of  mental  strategies  in  process  plant  diagnosis.  In  ].  Rasmussen  &  W.  B. 
Rouse  (Eds.),  Human  detection  and  diagnosis  of  system  failures  (pp.  241-258).  © 
1981  Plenum  Press  (Figure  4),  with  kind  permission  from  Springer  Science+Business 
Media  B.V. 


Third,  strategies  analysis  recognises  that  actors  often  shift  between  multiple  strategies  while 
performing  an  activity.  The  strategies  that  actors  adopt  at  any  given  moment  vary  as  a 
function  of  their  individual  preferences,  the  demands  of  the  situation,  and  the  resource 
requirements  of  each  strategy  such  as  the  amount  of  time,  memory,  or  knowledge  they 
require.  Accordingly,  by  identifying  the  design  requirements  of  multiple  strategies,  it  becomes 
possible  to  create  systems  that  provide  workers  with  the  flexibility  to  switch  seamlessly 
between  those  strategies,  depending  on  their  individual  preferences  or  judgment  as  to  which 
strategy  is  best  given  the  resource  requirements  and  situational  demands. 

Fourth,  strategies  analysis  is  concerned  with  identifying  the  set  of  strategies  that  is  possible, 
not  just  those  workers  use.  Workers  might  ignore  strategies  that  are  resource  intensive  but,  as 
a  consequence,  they  might  not  make  use  of  some  very  effective  strategies.  For  instance,  the 
hypothesis-and-test  strategy  described  earlier  was  not  favoured  by  electronic  technicians 
because  it  is  resource  intensive,  but  this  strategy  has  the  advantage  of  being  suitable  for 
dealing  with  unfamiliar  and  complex  faults.  The  resource  requirements  of  a  strategy  can  be 
manipulated  through  design,  for  example,  by  creating  effective  displays  or  offloading 
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demanding  aspects  of  a  strategy  to  automation.  Consequently,  workers  will  be  able  to  adopt 
strategies  they  otherwise  might  not  use. 

One  of  the  key  challenges  of  strategies  analysis,  then,  is  defining  the  set  of  strategies  that  is 
possible  for  any  given  system.  The  set  of  possible  strategies  goes  beyond  those  that  workers 
spontaneously  adopt  to  include  those  workers  could  adopt,  particularly  if  they  were  provided 
with  appropriate  support  (Figure  18).  Nevertheless,  the  strategies  that  workers  spontaneously 
adopt,  given  the  capabilities  or  constraints  of  the  human  cognitive  system,  are  an  essential 
subset.  A  key  source  of  information  for  this  subset  of  strategies  is  naturalistic  decision  making 
research.  For  example,  Klein's  (1989,  1998)  recognition-primed  decision  making  model 
describes  a  strategy  that  experts  use  for  decision  making  in  familiar  situations  and  under 
demanding  conditions  such  as  high  time  pressure.  This  strategy  has  been  found  to  be  relevant 
to  experts  in  a  range  of  systems  including  firefighting,  military,  and  medical  systems.  Thus, 
when  performing  strategies  analysis  for  the  same  systems,  a  significant  leverage  point  for 
analysis  and  design  is  provided  by  the  fact  that  the  recognition-primed  decision  making 
strategy  has  previously  been  studied  and  described  in  detail.  Accordingly,  with  the  in-depth 
study  of  functions  other  than  decision  making,  such  as  sense  making,  planning,  and  problem 
detection,  additional  leverage  points  are  possible  for  strategies  analysis.  This  is  one  reason 
that  the  recent  expansion  in  the  focus  of  naturalistic  decision  making  research  to  encompass 
functions  other  than  decision  making  is  of  significant  note. 


Strategies  workers 


if  provided  with 
appropriate  support 


Figure  18  The  set  of  possible  strategies  for  any  given  system  includes  strategies  that  workers 
spontaneously  adopt  as  well  as  strategies  they  could  adopt,  especially  if  they  were  provided 
with  appropriate  support. 


9.  Conclusion 


In  conclusion,  although  much  has  been  achieved  with  cognitive  work  analysis  over  the  last 
few  decades,  much  remains  to  be  done.  Based  on  empirical  studies  of  how  work  is  achieved  in 
naturalistic  settings,  Jens  Rasmussen  and  his  colleagues  have  provided  us  with  a  solid 
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foundation  for  work  analysis.  This  framework,  which  focuses  on  examining  the  possibilities 
for  work  within  a  set  of  boundaries  or  constraints,  promotes  designing  for  adaptation 
(Rasmussen  et  al.,  1994;  Rasmussen,  1986). 

The  EID  framework  (Rasmussen  &  Vicente,  1989;  Vicente  &  Rasmussen,  1990, 1992),  which 
was  formulated  from  the  concepts  of  cognitive  work  analysis,  has  been  subjected  to  a 
comprehensive  program  of  controlled  experimental  investigations.  This  research  has 
demonstrated  that  EID  can  lead  to  better  performance  than  that  supported  by  existing 
interfaces  for  a  range  of  systems.  The  EID  program  has  addressed  in  part  the  limitations  of  the 
original  research  by  the  Riso  group  relating  to  the  lack  of  experimental  testing  of  concepts  and 
the  generalisability  of  those  concepts  to  a  variety  of  systems.  Future  work  on  EID  will  include 
extensions  to  this  framework  to  incorporate,  among  other  things,  the  other  dimensions  of 
cognitive  work  analysis  as  well  as  other  work  analysis  techniques. 

Novel  applications  of  cognitive  work  analysis  to  problems  other  than  interface  design  have 
been  achieved  largely  in  industrial  settings.  Some  of  these  applications  involve  extensions  to 
cognitive  work  analysis  to  encompass,  for  example,  the  development  of  new  team  or 
organisational  designs.  These  applications  have  had  an  impact  in  industrial  settings  and  have 
made  a  unique  contribution  to  design  relative  to  that  of  standard  techniques,  but  empirical 
evaluation  is  lacking.  For  this  type  of  evaluation  to  be  achieved,  a  shift  to  a  research  setting 
may  be  necessary. 

The  methodological  program  on  cognitive  work  analysis  complements  the  seminal 
monographs  by  Rasmussen  et  al.  (1994)  and  Vicente  (1999),  which  focus  on  providing  a 
comprehensive  conceptual  perspective  of  this  framework.  While  a  substantive  start  has  been 
made  on  methodology,  much  more  work  is  necessary,  especially  in  relation  to  the  dimensions 
subsequent  to  work  domain  analysis. 

The  future  program  on  the  subsequent  dimensions  will  echo  the  historical  development  of 
cognitive  work  analysis  by  building  on  the  research  of  a  range  of  communities.  Just  as  the 
work  of  Rochlin  et  al.  (1987),  for  instance,  had  significant  impact  on  the  concepts  of  cognitive 
work  analysis,  the  insights  of  other  researchers,  including  those  in  the  naturalistic  decision 
making  community,  may  be  influential.  A  significant  challenge  for  researchers  will  be  to  build 
on  any  insights  in  a  way  that  remains  faithful  to  the  theoretical  underpinnings  of  cognitive 
work  analysis,  which  appears  to  offer  a  potent  approach  to  design. 

These  achievements  in  cognitive  work  analysis  are  likely  to  spark  significant  interest  in  this 
framework  over  the  next  decade.  This  is  already  becoming  evident  in  the  growing  number  of 
publications  on  cognitive  work  analysis  over  the  last  few  years,  many  by  researchers  and 
practitioners  who  are  new  to  the  area.  The  immediate  result  will  be  a  proliferation  of 
applications  of  cognitive  work  analysis,  including  many  novel  applications.  Eventually,  then, 
one  of  the  biggest  challenges  for  cognitive  work  analysis  will  be  to  support  an  integrated 
approach  to  system  design. 

Designing  for  adaptation  is  a  complex  problem  that  will  not  be  resolved  in  a  piecemeal 
fashion.  It  will  not  be  resolved  by  just  an  ecological  interface,  as  Vicente  (2002)  recognised, 
and  as  I  have  experienced,  it  will  not  be  resolved  by  just  a  team  design.  Instead,  all  elements  of 
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a  system's  design  must  be  systematically  integrated  in  a  way  that  explicitly  supports  this 
objective,  which  raises  theoretical,  methodological,  and  practical  challenges.  I  believe  that 
cognitive  work  analysis  has  the  potential  to  meet  these  challenges,  but  the  task  will  not  be 
simple. 
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